########################################################################
#  scanner - scanf with multiple line template and regex
#  Copyright (C) 2007,8 Ehud Ben-Reuven
#  udi@benreuven.com
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation version 2.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
########################################################################
""" Scan input text file using a template file.
Useful to extract information from flat, line oriented text files that differ in
well defined locations from a reference template file (e.g. a bank statement.)
The template defines
all the parts that are expected to change, extract some of them and validty that
nothing else has changed. This is similar to the C scanf function that works on
a single line and is somewhat the reverse of embedded languages that allows you
to do small changes to a template file.

The goal of this routine is to simplfy the template as much as possible.
When creating a template start with an example of an input file, keep as-is the
parts of the file that are expected to remain fixed (e.g. headers and footers)
and replace the parts that are expected to change with a regular-expression.
For example, suppose the input is something like:

'"Junk" total on 9/25/06 is *** 12345 *** change 1.2%'

and the only part that changes is the 5 digit total value.
The template can be:

'"Junk" total on 9/25/06 is *** @"....."@ *** change 1.2%'

The expression is surronded by '@' which are removed and the remaining string is
evaluated as a Python expression which eats the two double-quotes.
(if you want to use '@' character for other usages escape it by adding '\' before it)
The result of the evalution should be a regular expression, in this case  '.....',
which will match any 5 characters.
If you want to limit the template to 5 digits you can use the following:

'"Junk" total on 9/25/06 is *** @"\\d{5}"@ *** change 1.2%'

Where '\\' is used because one Python evalution process, and '{5}' is part of Python's regular expression syntax.
You can have any number of Python expressions in the same line. For example:

'"@".*?"@ total on 9/25/06 is *** @"\\d{5}"@ *** change 1.2%'

Each expression can have in it named groups and the result of their match
is kept in a list of dictionaries which is returned by this function.
For example, the template:

'"Junk" total on 9/25/06 is *** @"(?P<mytotal>\\d+)"@ *** change 1.2%'

will give: [{'mytotal': '12345'}]
However, there is one limitation, the entire template line is first converted 
into a single regular expression so it is not possible to use the same group 
name in more than one expression in the same line, also the usage of numbered 
group can be confusing and should be avoided. To help with this you can use 
group names that begin with an underline ('_') these group names are temporary 
and will not appear in the final list of dictonaries.

The namespace used when evaluting the Python expression comes pre-loaded with 
several functions that can be used to simplfy the template.
For example, the template:

'@s('name')@ total on @date('when')@ is *** @d('mytotal')@ *** change @f('delta')%'

will give: [{'name': 'Junk', 'when': '9/25/06', 'mytotal': 12345, 'delta': 1.2}]
Note that the value for 'mytotal' is an integer and for 'delta' is a float.

You can add your own functions by adding them to namespace variable.
For example:

namespace["mytotal"]=lambda : '(?P<mytotal>\\d{5})'

and then using it in a template

"Junk" total on 9/25/06 is *** @mytotal()@ *** change 1.2%'''

Will ensure that total is made of exactly 5 digits and otherwise the entire 
match will fail. The resulting dictonary will show {'mytotal': '12345'}
Note that the value in the dictionary is a string. This can be solved by adding 
another function
called 'post_mytotal' that will receive as input the string matched to 'mytotal':
Adding to the above example:

namespace["post_mytotal"]=lambda s: int(s)

will give the dictonary entry {'mytotal': 12345}
For more ideas, see below how namespace is built and the examples in the main part.

Up to now we talked about a single line from the template compared with a single
line from input. Both can have more lines: every line in template is converted
to one regular expression and is made to match the next input line.
Note that expressions surronded by '@' must start and end in the same template 
line. Each succssful match returns a dictonary of group names and their values,
and the entire dictonary (if not empty) is appended to a list which is returned
by this function. As an aid, the  entry 'nline' which keeps track of the line 
number is added to each dictonary. The entire match process fails if any of 
these matches fails, and the function returns None. For example, the input:

'''Line#,Voltage
1,12.2
2,23.4
3,4'''

can be scanned with the template:
    
'''Line#,Voltage
@d('_index')@,@f('value')@
@d('_index')@,@f('value')@
@d('_index')@,@f('value')@'''

which will give the following list of dictonaries:
[{'nline': 1, 'value': 12.2}, {'nline': 2, 'value': 23.4}, {'nline': 3, 'value': 4.0}]
Note that the first line (nline=0) is matched, but since it does not have any group names,
its dictonary was empty and therefore it did not appear in the final list.
Also note that the first column is validated to be an integer, but,
its value does not appear in the result since '_index' begins with an underline.

In the above example the input contains a table made from several similar lines.
In this case a single template line can be used by placing a '*' at the begining
of the template line. The same template will be used to match one line after the
other from the input until its end or until the match will fail,
in this case instead of failing the entire match process,
the next line from the template file will be attempteted.
For example, the input:

'''Line#,Voltage
1,12.2
2,23.4
3,4
Total 39.6'''

can be scanned with the template:
    
'''Line#,Voltage
*@d('_index')@,@f('value')@
Total @f('total')@
'''

"""

import re

class scannerror(Exception):
    """ Failed to match tempate with input. """
    def __init__(self, i_nline,t_nline,inline,r):
        """ A value of -1 indicates End-Of-File """
        self.i_nline = i_nline
        self.t_nline= t_nline
        self.inline=inline
        self.expression=r
    def __str__(self):
        return 'Failed to match on line %d of input and line %d of template\n%s\n%s\n'%(self.i_nline, self.t_nline,self.inline,self.expression)

def scanner(template,input):
    # Generate a list of regular expresions lines out of the template
    txtreg = []
    # For each line keeps a modifier: NORMAL_LINE - match one line, MULTI_LINE - match zero or more lines
    NORMAL_LINE, MULTI_LINE, EXT_LINE = 0,1,2
    txtmod = []
    try:
        template_iter=template.readline # TODO: does not work as an iterator
    except:
        template_iter=template.splitlines
#    for t_nline, l in enumerate(template.splitlines()):
    for t_nline, l in enumerate(template_iter()):
        txtreg_l=[]
        if l.startswith('*'):
            l=l[1:]
            txtmod.append(MULTI_LINE)
        elif l.startswith('&'):
            l=l[1:]
            txtmod.append(EXT_LINE)
        else:
            txtmod.append(NORMAL_LINE)
        # Within each TEXT line, '@' is used to delemite python expressions
        lsegments = re.compile(r"\@(?<!\\)(.*?)\@(?<!\\)").split(l)
        for j, lsegment in enumerate(lsegments):
            if not lsegment: continue
            if j % 2:
                try:
                    txtreg_l.append(eval(lsegment,namespace))
                except SyntaxError, v:
                    v.lineno = t_nline
                    raise
            else:
                lsegment = lsegment.replace("\\@","@")
                lsegment = re.escape(lsegment)
                txtreg_l.append(lsegment)
        txtreg.append("^" + "".join(txtreg_l) + "$")

    # match every line in txtreg with every line in input
    # A line in txtreg that begins with a '*' is used for as many lines as it can in input and
    # when it fails to match, the next line in txtreg is used
    t_nline=0
    output=[]
    multimode=False
    t_multinline=-1
#    input=input.splitlines()
#    for i_nline, inline in enumerate(input):
    try:
        input_iter=input.readlines()
    except:
        input_iter=input.splitlines()
    for i_nline, inline in enumerate(input_iter):
        while True:
            # check if there are more input lines than template lines
            if t_nline >= len(txtreg):
                raise scannerror(i_nline,-1)
            r=txtreg[t_nline]
            extmode=False
            if multimode and txtmod[t_nline]==EXT_LINE:
                extmode=True
            elif txtmod[t_nline]==MULTI_LINE:
                multimode=True
                t_multinline=t_nline
            else:
                multimode=False
            
            found=re.match(r,inline)
            if not found:
                if multimode:
                    if t_nline==t_multinline:
                        # if we are on the first line of a multi line segement and we failed then we should skip the multi line segment
                        multimode=False
                        while True:
                            t_nline+=1
                            if t_nline >= len(txtreg) or txtmod[t_nline]!=EXT_LINE: break
                        continue
                    elif t_nline < len(txtreg)-1 and txtmod[t_nline+1]!=EXT_LINE:
                        # if we are at the last line of a multi line segment then perhaps we should start a new match with the entire multiline segment
                        r=txtreg[t_multinline]
                        found=re.match(r,inline)
                        if not found:
                            multimode=False
                            t_nline+=1
                            continue
                        else:
                            t_nline=t_multinline
                    else:
                        raise scannerror(i_nline,t_nline,inline,r)
                else:
                    raise scannerror(i_nline,t_nline,inline,r)
            linedict=found.groupdict()
            if linedict:
                # ignore names that start with '_'
                linedict = dict([(name,linedict[name]) for name in linedict if not name.startswith('_')])
                # For every name found, check if there is a post-processing function
                for name in linedict:
                    try:
                        post=namespace['post_'+name]
                        linedict[name]=post(linedict[name])
                    except KeyError:
                        pass
                linedict["nline"]=i_nline
                output.append(linedict)
            t_nline+=1
            if multimode and (t_nline >= len(txtreg) or txtmod[t_nline]!=EXT_LINE):
                t_nline=t_multinline
            break
    # check if we have leftover in txtreg that should have been matched
    if t_nline < len(txtreg):
        if multimode and txtmod[t_nline]==EXT_LINE:
                raise scannerror(-1,i)            
        for i in range(t_nline,len(txtreg)-1):
            if txtmod[i]==NORMAL_LINE:
                raise scannerror(-1,i,'EOF',txtreg[t_nline])
    return output

def f(n):
    # Define the post processing for a float number: casting it from string to float.
    namespace['post_'+n] = lambda s: float(s)
    # Define a regular expression for float and embedd the name of the group
    return '(?P<%s>[+-]?\d+(\.\d*)?)'%n

def d(n):
    # Define the post processing for a int number: casting it from string to int.
    namespace['post_'+n] = lambda s: int(s)
    # Define a regular expression for float and embedd the name of the group
    return '(?P<%s>\d+)'%n


namespace={\
    "s": lambda n: '((?P<_%s>\\")|(?!\\"))(?P<%s>.*?)((?(_%s)\\"|(?<!\\")))'%(n,n,n),\
    "d": d,\
    "f": f,\
    "date": lambda n: '(?P<%s>(\d|1[0-2])/\d{2}/\d{2})'%n,\
       }

# Bellow is a list of examples of how to use the program.
if __name__=='__main__':
    input = r"""1,2

3,4

"""
    
    template = r"""*@'\d'@,@'\d'@
&"""

    try:
        print scanner(template,input)
    except scannerror, v:
        print v

    input = 'Hello world'
    template = 'Hello world'
    print scanner(template,input)
    
    input = r'''
Hello world
1,2
3,4
5,A
'''.strip()
    
    template = r'''
Hello world
*@'\d'@,@'\d'@
'''.strip()
    
    try:
        print scanner(template,input)
    except scannerror, v:
        print v

    template = r'''
Hello world
*@'\d'@,@'[a-zA-Z\d]'@
'''.strip()
    
    try:
        print scanner(template,input)
    except SyntaxError, v:
        print v

    input = r'"Junk" total on 9/25/06 is *** 12345 *** change 1.2%'
    
    namespace["post_mytotal"] = lambda s: int(s)
    namespace["foo"] = lambda : '(?P<mytotal>\d{5})'

    template = r'''
@s('name')@ total on @date('when')@ is *** @foo()@ *** change @f('delta')@%
'''.strip()
    print scanner(template,input)

    input = r"""
Line#,Voltage
1,12.2
2,23.4
3,4
Total 39.6
""".strip()
    
    template = r"""
Line#,Voltage
*@d('_index')@,@f('value')@
Total @f('total')@
""".strip()
    
    print scanner(template,input)
    
    # Example of stripping the header out of a file and printing the rest
    template=r"""
Line#,Voltage
*@s('line')@
""".strip()

    def print_line(s):
        print s
    namespace["post_line"]=print_line
    scanner(template,input)
